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ABSTRACT 

The selection of courseware is a difficult problem in 
the use of computers in education. A dissemination procedure for a 
courseware selection instrument involving workshops for computer 
coordinators is described. Computer coordinators of 267 Dutch 
secondary schools were asked how they used certain sources of 
information in selecting courseware. Many used results of evaluations 
by the Soft- and Courseware Evaluation Centre for the Netherlands, 
but expressed the need for additional information. Six secondary 
schools participated in a workshop for computer coordinators who were 
taught to use an instrument called "The Courseware Selector" to 
evaluate courseware. The coordinators were expected to disseminate 
the information in their schools through further workshops. A 
multiple case study design was used to compare developments in the 
six schools involved in this cascade approach. In general, effects on 
the overall selection of courseware were not large, but promising 
effects did appear related to teachers becoming better acquainted 
with courseware and reducing their computer phobia. A critical 
variable in implementation was the level of experience in computers 
at the school initially. (Contains 7 references.) (SLD) 



****************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document . * 

************************************************* ********************** 



U.S. Of MWmMMT OF EDUCATION j 

Often 01 £duc*wws* R««»«fcn and imOWtmtw 
EDUCATIONAL RESOURCES INFORMATION 

CENTER <ER*0 
O Thii document r*a own r#pro4t*c«d as 

r*c*v«d trom th« p#fton or ofgtnutlion 

ortQtnaiino «t 
O Minor cntn^t nav« »*«d« to imorov* 

r •production gustily 



• Po»nitoU*wo*OOintOfltHtt«<3inlhiadocu- 
m«m do not r*c«m»»»v r«or»t»nt otticiti 
OERI portion or policy 



A PROCEDURE FOR SUPPORTING THE SELECTION OF COURSEWARE BY 
<j> SECONDARY SCHOOL TEACHERS 

GO Baukje J. van Kesteren, Gerrit J. Carleer, 

£3 Betty A. Collis, and Tjecrd Plomp 



Paper presented at the European Conference on Educational Research 
Enschede (The Netherlands), 22-25 June, 1992 



^ BEST COPY AVAILABLE 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



1 Gerrit J. Carleer 

2 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)/' 



1 



A PROCEDURE FOR SUPPORTING THE SELECTION OF COURSEWARE BY 

SECONDARY SCHOOL TEACHERS 



Baukje J. van Kesteren 1 , Gerrit J. Carleer 2 , Betty A. Collis 2 , and Tjeerd Plomp 2 



One of the main problems regarding the use of computers in education is the selection of 
courseware. For several reasons, courseware that may be eventually bought is difficult to 
access. In addition, there are difficulties relating to the support of the computer coordinator 
as the conduit through which preliminary information about courseware must flow as it 
arrives in the school. This study describes a dissemination procedure for a courseware 
selection instrument involving a workshop for computer coordinators and similar workshops 
(offered by the thus-trained coordinators) for their fellow-teachers. Results from six secondary 
schools and the implications of these results for the overall development of a framework and 
an instrument for courseware selection are described. 

Introduction to the Problem 

The value of the use of computers in the classroom is influenced by the quality of the 
courseware that is being used. By 'courseware' is meant educational software accompanied by 
other materials such as work sheets for the students or a guide for the teacher. However, for 
teachers it is not a simple task to make an accountable choice of courseware that may be 
eventually bought. Difficulties confronting the selection of courseware occur at various levels. 

Preceding the present study, in the autumn of 1989 the computer coordinators of 267 
Dutch secondary schools were asked in which measure they used certain sources of 
information while selecting courseware, which characteristics of a courseware package were 
thought important, and which problems they experienced regarding the selection of 
courseware. Although 220 computer coordinators in this pre-study claimed to use the results 
of evaluation studies published by the Soft- and Courseware Evaluation Centre for the 
Netherlands (SCEN) and 247 said they used descriptions in magazines of packages by persons 
experienced with the packages always or incidentally, 92 computer coordinators had been 
confronted with the problem that courseware packages that had been selected did not answer 
expectations when used in the classroom. Apparently, information about a package is not 
always sufficient to make the right choice. As was noted, these experiences were had by 
computer coordinators, teachers who are usually more informed about computers and 
courseware than most of their fellow-teachers. It may be further supposed that teachers do not 
always have at their disposal all the information that is available to the computer coordinator 
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in the school Besides, this information is often too broad and insufficiently aimed at the 
needs of the teachers who want to know whether a package is specifically useful for their 
pupils. 

The conclusion is that there is an important need for strategies to provide teachers with 
better information about courseware and with some sort of aid to help them form a better 
perspective on the usefulness of different courseware alternatives. During the present research 
and implementation study such a strategy was developed of which an aid called the 
'Courseware Selector' (Carleer, in the press) formed a part. 

Method 

Theoretical framework 

A theoretical framework based on the work of Fullan (1982) and of Fullan, Miles, and 
Anderson (1988) supported the selection of relevant variables affecting the effectiveness of 
courseware selection strategies and the possible relations between these variables. For this 
purpose it was necessary to distinguish between what was called the 'large innovation', that 
is, the use of computers in the school, which was already in process during the years 
preceding our study, and the 'small innovation', the strategy for the selection of courseware 
that was the focus of the project. Fullan (1982) mentions 15 factors influencing the 
implementation of innovations in general. Fullan, Miles and Anderson mention 10 factors that 
are especially important in the implementation of the use of computers in schools. Not every 
factor mentioned by these authors was expected to be relevant to our study or to the Dutch 
situation, respectively. Thus, some factors mentioned by Fullan or by Fullan, Miles and 
Anderson have been combined, other factors have in a certain sense been translated to suit the 
situation. In all, nine factors of which some were thus adapted, were expected to influence the 
implementation of the 'large innovation'. These factors are: 

1. The state of affairs regarding the large innovation: this factor was composed of Fullan's 
factors 'clarity and complexity', 'quality and practicability', 'process for implementation 
and institutionalization', and 'the adoption process' of the large innovation,; 

2. Consensus/conflict about the change in connection with the large innovation; 

3. Professional development and assistance in connection with the large innovation; 

4. Implementation monitoring and problem solving in connection with the large innovation; 

5. Principal's leadership (partially delegated to a steering committee for the large innovation 
or the computer coordinator); 

6. Environmental stability; 

7. Teacher-teacher relations in connection with the large innovation; 

8. Teacher characteristics and orientations; 

9. The history of former innovative attempts. 

Ten factors were expected to influence the implementation of the 'small innovation', 
namely: 

1. Clarity and complexity of the small innovation; 

2. Consensus/conflict about the change in connection with the small innovation; 

3. Quality and practicality of the change in connection with the small innovation; 

4. The planning of the process for the implementation and the institutionalization of the small 
innovation; 

5. Professional development and assistance in connection with the small innovation; 
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6. Implementation monitoring and problem solving in connection with the small innovation; 

7. Teacher-teacher relations in connection with the small innovation; 

8. Teacher characteristics and orientations; 

9. The adoption process of the small innovation; 

10. Expectations with respect to the project as a whole. 

The implementation of the small innovation was in turn expected to influence some of the 
19 abovementioned factors. Thus, besides being influencing factors, these were in turn 
transformed into output variables. The phenomena at hand are: 

1. Consensus/conflict about the change in connection with the large innovation; 

2. Professional development and assistance in connection with the large innovation; 

3. Principal's leadership (partially delegated to a steering committee for the large innovation 
or to the computer coordinator); 

4. Teacher-teacher relations in connection with the large innovation. 

5. Teacher characteristics and orientations; 

6. Clarity and complexity of the small innovation; 

7. Consensus/conflict about the change in connection with the small innovation; 

8. Quality and practicality of the change in connection with the small innovation; 

9. Professional development and assistance in connection with the small innovation; 

10. Implementation monitoring and problem solving in connection with the small 
innovation; 

11. Teacher-teacher relations in connection with the small innovation. 

To evaluate the implementation of the small innovation, criterion variables were regarded to 
be: 

1. Feelings of competency and satisfaction of the computer coordinators regarding the process 
of courseware selection; 

2. Feelings of competency and satisfaction of the teachers regarding the process of 
courseware selection; 

3. The evaluation of the courseware selection project by the computer coordinators; 

4. The evaluation of the courseware 5 'lection project by the teachers. 

Selection of participating schools 

Out of a group of about 1000 secondary schools which had participated in a national 
project supplying them with hard- and software (the NIVO-project), some 320 schools were 
sent a questionnaire regarding the use they made of courseware and asking whether they were 
willing to participate in the present project on courseware selection. Out of these 320 schools 
85 returned the questionnaire, and among these 85, 26 were willing to participate. 

T.iree criteria had been formulated for the participation of the schools in the project: 

1. They belonged to the first, second or third group of schools which took part in the 
abovementioned NIVO-project. This implied that they had had computers at their disposal 
for several years. 

2. There was a computer coordinator in the school; this implied that there was someone who 
already served as an intermediary between the information about courseware that is given 
in magazines and official publications and the teachers who are supposed to use the 
courseware. 

3. At least two teachers out of two different sections each were willing to participate in the 
project. 

Of the 26 schools which were ready to participate, only 8 met criteria numbers 2 and 3. 



Out of these 8 schools, 6 were selected on the practical basis of being within easy reach of 
the research team. 

Workshop for computer coordinators 

Based on the analysis described under Theoretical framework' an inservice workshop for 
computer coordinators was developed as the central component of a dissemination procedure 
for better courseware selection. 

The first half of the workshop was spent on information to the computer coordinators 
about different ways of organizing computer use in schools. This was done by two 
experienced computer coordinators who were taking part in a separate longitudinal experiment 
on the implementation of computer use in secondary schools. They gave information about 
the way in which the implementation of computer use was organized within their schools, 
about the problems they had met and the solutions they had found. 

During the afternoon, the computer coordinators were instructed in the use of an instrument 
for the selection of courseware by teachers called the 'Courseware Selector', and in strategies 
for the organization of a workshop on courseware selection for their fellow-teachers. The 
Courseware Selector consists of three parts, the first part guiding the search for existing 
courseware, the second part guiding the examination of the information accompanying a given 
courseware package, and the third part guiding the examination of the courseware itself. 
These guides have the form of questionnaires. The first part (regarding the search for existing 
courseware) consists of a synopsis of sources in which information about courseware can be 
found (descriptions, evaluations, users' experiences, etcetera), such as magazines, publications 
by institutions like SCEN, NICL (Dutch Information Centre for Educational Equipment), the 
bulletin board of the SLO (Foundation for Curriculum Development) and on-line data banks. 
The second and the third part of the Courseware Selector exist of questionnaires that are to be 
filled out by the teachers, in order to help them focus on the relevant aspects of the packages 
in their own special situation. 

Examples of questions in the second part (regarding the information accompanying a 
courseware package at hand) are: 

- Does the software work on your school's computers? 

- What is the target group of students? 

- Does the topic fit in your own curricular and instructional practice? 

- Is the package related to certain textbooks? 

Examples of questions in the third part (regarding the courseware itself) are: 

- Is the courseware practical from a didactical and/or classroom-organization perspective? 

- For what kinds of student exercise does the courseware allow? (For example, knowledge 
reproduction, knowledge application, acquisition of new knowledge and comprehension.) 

- What prerequisite knowledge or previous experience do students need to work with the 
program? 

After having studied the material and having answered the questions on the topics above, 
participants in individual subject areas are urged to get together and consult with each other 
on their findings and on the eventual decision about the purchase of the package. 

Multiple case study; data collection 

A multiple case study design was developed to compare developments in the six secondary 
schools involved in this cascade approach. At the start of the study information was gathered 
by interviewing the computer coordinators and one of the school leaders of each school, and 
by means of questionnaires from the participating teachers. The information related to the 
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extent to which computers were used in each school and for what purpose; the attitudes of the 
teachers towards the use of computers in the classroom (measured by an adapted version of 
the Concerns Based AdbftJo*Modt«L or CBAM Hall, George, & Rutherford, 1977); the 
process ot coiuseware stieetkm oy Teacneis in eacn school up to that point; the ways in which 
teachers had been educated for computer rse; their satisfaction with courseware that had been 
purchased up to that time; their feelings of competency in selecting courseware; aspects of 
courseware that were considered important; and the expectations of the participating teachers 
and computer coordinators for the courseware selecting project At the end, information was 
collected on the effects of the inservice workshop for the computer coordinators, the impact 
of this on subsequent in-school workshops about courseware selection organized by the 
computer coordinators for their fellow-teachers, and to the sort of procedures used for 
courseware selection by teachers at the end of the study. In between, the workshops which 
were organized by the computer coordinates for their fellow-teachers were also observed and 
diaries were collected in which the computer coordinators had registered their activities 
relating to the use of computers in their schools. Particular consideration was given to 
changes in the use of computers in the schools in general, and in particular, to the selection of 
courseware as it took place in the schools. 

These data were mostly qualitative in nature, although within each case quantitative data 
were also gathered, during the interviews at the start and the end of the study and in the 
questionnaires for the participating teachers. 

Data analysis 

The sets of data were analyzed by seeking patterns in the material, much in the way Yin 
(1984) advises with respect to the formulation of 'case study questions'. Such questions were 
developed for this study, and related to the situation in the schools at the start of the study, 
the implementation of the use of the Courseware Selector, the effects of the project as a 
whole, and the conclusions that were to be drawn from the overall experience. Examples of 
these case study questions include the following. 
Concerning the situation at the start of the project: 

- How can the use of computers in this school in general be characterized? 

- On which information is this characterization based? 

- By which further information is this characterization supported? 

- Which possible information is at variance with this characterization? 
Concerning the conclusions to be drawn from the overall experience: 

- To what extent can one conneci the positive or negative developments, or the absence of 
further developments during the project to the factors that have been described at the start 
of the project? 

- To what extent is success or failure a consequence of the strategy for courseware selection 
that was offered? 

- To what extent is success or failure a consequence of the way in which the computer 
coordinator put into practice the strategy for courseware selection? 

At the completion of the study, the findings were linked to the theoretical framework. 



Results 

The results consist of reports about each separate participating school. In these reports the 
situation of each school with respect to the use of computers in and outside of the classroom 
is described, as well as the organization of computer use, the organization of courseware 
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selection, the implementation of the strategy for courseware selection which was offered 
during the project, what the computer coordinator had learned from the workshop about 
organizational aspects of computer use and what he had put into practice, to what extent the 
workshop had been satisfying and had influenced the procedures of courseware selection in 
the school, and to what extent the expectations of the computer coordinators and the teachers 
of the project had been met. . 

The results differ from school to school, but the main effects can be summarized as 
follows. Although the effects of the overall strategy for improving the selection of courseware 
varied between schools, these effects were in general rather disappointing. Instead of 
enhancing the teachers' skills in selecting courseware, the main effect of the workshop that 
was offered to them by their computer coordinators was increasing their enthusiasm for the 
use of courseware. Although increased enthusiasm is not an undesirable effect in itself, it was 
not the effect that was most immediately sought. The main reason for an apparent lack of 
change in teachers* skill levels may be the fact that the use of computers in Dutch classrooms 
is generally occurring at such a low level that strategies to improve the selection of 
courseware do not have an adequate foundation. This seemed to be the case even in the six 
schools that were selected for the study; schools that were comparatively far in computer-use 
experience based on national comparisons. 



Conclusions 

The main conclusions are as follows: 

1. The level of experience in the school relevant to computer use at the start of the 
implementation of a strategy for courseware selection is a critical variable in the success of 
subsequent implementation strategies. Schools or sections in which teachers have not yet 
used or selected courseware are not yet prepared for a strategy that focuses on the 
effective selection of courseware. 

2. The information about alternate strategies for the organization of the use of computers in 
schools that was given to the computer coordinators had different effects on the different 
coordinators. To those who felt relatively insecure about supporting computer use in their 
schools at the start of the project, this part of the course offered new insights about how to 
stimulate the use of computers by their fellow-teachers. In these cases, strengthening the 
functioning of the computer coordinator appeared to improve the results relating to the 
implementation of the overall strategy for courseware selection. 

3. In general, the effect of the overall strategy for the selection of courseware on teachers* 
intentions relative to courseware selection and use in the future was not very large. 
However, promising effects that did appear related to teachers getting better acquainted 
with courseware, increasing their enthusiasm for using courseware, and lessening their 
levels of 'computer-phobia'. 



Back to the theoretical framework 

Linking the conclusions mentioned above to the factors mentioned in the paragraph 
entitled 'Theoretical framework', leads to the following: 

- The state of affairs regarding the large innovation generally was not yet advanced enough 
for a successful implementation of the small innovation. 

- The 'quality' of the small innovation was generally judged positively. However, one aspect 
of 'practicability' of an innovation is the question of whether it meets important needs of 
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pupils and (or) teachers. For many teachers the need of a strategy for courseware selection 
appeared to be not yet under discussion, because they did not have enough experience with 
the use and the selection of courseware. 

- The planning of the 'process for implementation and institutionalization' of the small 
innovation played a role, too. The extent to which the teachers intended to select 
courseware in future with the aid of the Courseware Selector appears to be related to the 
way in which the workshop had been offered to them. This implementation process, 
however, in turn was dependent upon the state of affairs of the large innovation. 

- The factor 'professional development and assistance' regarding the large innovation mainly 
played a role as an indicator of succes for the extent to which the large innovation had 
advanced. The same holds for 'principal's leadership'. 

- The 'characteristics and orientations' of the teachers played an important role. Their scores 
on the CBAM at the start of the project show that they were in a relative early stage as far 
as their experiences with computer use in the classroom. As was said, this was seen as the 
most important reason why the implementation of the small innovation only partially 
succeeded. 

- Although not mentioned by Fullan, the expectations of the computer coordinators and the 
teachers had been taken up as a factor in the design. As far as the computer coordinators 
had positive expectations, these generally were met. In many cases the teachers' 
expectations were not realistic in the sense that they did correspond with the goals of the 
project. Some 'real', positive, expectations were met, some negative ones were, too. 

Not all factors which were mentioned in the theoretical framework appear to have explicitly 
played a role in the implementation of the small innovation. 

- 'Consensus or conflict' about the large or the small innovation does not seem to have 
played an important role; maybe because there was no conflict. 

- 'Monitoring the implementation' and 'problem solving' regarding the large and the small 
innovation had not been explicitly observed by the researchers. 

- The same holds for the 'stability of the environment', perhaps because the environment of 
all participating schools was relatively stable. 

- Experiences with former innovations do not appear to have played a large (positive or 
negative) role in the implementation of the small innovation. 

- The 'adoption process' of the small innovation was also not under observation. This 
became clear especially with the question as to what extent teachers were interested in the 
workshop about courseware selection. In one case, participating in the workshop by the 
teachers was compulsory. As far as we know, in most cases participating by the teachers 
was optional, but our insight in the question about how enthusiastic they were was very 
limited. 

Under the paragraph 'Theoretical framework' some factors were mentioned which in turn 
were expected to be influenced by the implementation of the small innovation. At the end of 
the courseware selection project it was not yet possible to tell to what measure most of these 
'secondary output variables' had been influenced by the implementation. However, three 
factors arise about which it is possible to speculate. 

- 'Professional development and assistance' regarding the small innovation. At some schools 
this has been reinforced, e.g., by establishing a computer room for teachers, by (plans for) 
more workshops about courseware selection, or by (plans for) a visit by a section to a 
'regional point of support' regarding information technology in schools. 

- 'Teacher-teacher relations'. All teachers who participated in one of the workshops about 
courseware selection and who also filled out the questionnaire at the end of the study 
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stated that they intended in the future to consult together about the purchase of courseware. 

- 'Characteristics and orientations of teachers'. Comparing the mean CBAM-profiles within 
schools at the start and the conclusion of the study shows that the maximum of the profile 
had been altered in a way that can be interpreted as showing that the teachers' orientations 
regarding the use of computers in the classroom had changed during the project in a more 
positive direction. 

Discussion and recommendations 

The dissemination strategy for the use of the Courseware Selector that was used in this 
study is more effective in those cases in which teachers already have used and, preferably, 
have selected courseware before. 

A workshop similar to the one which was offered in this study may be useful in schools 
which are less advanced in the use of computers, but the contents of this workshop should 
relate to the use of computers in the classroom instead of the selection of courseware. 

The decision to call in the computer coordinators to spread information and to teach skills 
regarding the selection of courseware appears to be a good one. 

In the final report recommendations were given to schools regarding the organization of 
(aspects of) the use of computers in the school and in the classroom in general, and the 
organization of a workshop about courseware selection in particular. The main 
recommendations are as follows: 

- Schools are recommended to organize a workshop about courseware selection only for 
sections in which courseware has been already used and in which some teachers already 
have selected courseware. 

- The person who conducts the workshop must be well instructed about the second and the 
third parts of the Courseware Selector. The second part (the one guiding the examination 
of print material accompanying a software package) contains questions that cannot always 
be answered on the grounds of the available documentation. This is important because it 
makes clear which information is lacking after reading the manual; however, if the 
participants are not informed about this possibility, this might provoke irritation. 

- Workshops about courseware selection as well as workshops aiming at getting acquainted 
with courseware are best produced for complete subject area sections. One reason is that 
consultation within a section plays an important role in the use of the Courseware Selector. 
A second reason is that joint activity in the field of computer use appears to have a 
stimulating effect on teachers. 

- It is important that a workshop about courseware selection is not 'overcrowded'. A 
workshop is most effective when every participator has a computer at his or her disposal 
so that computer packages can be viewed individually and that questions about the 
program can be answered individually. 

- The computer coordinators in this study experienced support and stimuli from the detailed 
guide which was offered for the workshop and from their fellow-computer coordinators. 
Therefore, a recommendation to, among others, institutions for support and in-service 
training is to provide computer coordinators with means and instruments to enhance the 
use of computers and courseware, and to create possibilities for contact among computer 
coordinators. 

- Other conditions and stimuli for enhancing the use of computers (in the classroom) are: 
- enough released time for the computer coordinator; 
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- a computer room for teachers where they can use a computer without students observing 
them; 

- a well-working circulation system for information about courseware that might be 
purchased or that has recently been purchased; 

- the presence of a 'technical educational assistenf who can assist with the use of 
computers in the classroom. 
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